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Abstract 


Two  important  aspects  of  part  inspection  are  the  measurement  of  surface  shape  and  surface 
roughness.  We  propose  a  noncontact  method  of  measuring  surface  shape  and  surface  rough¬ 
ness.  The  method,  which  we  call  “four  light  photometric  stereo”,  uses  four  lights  which 
sequentially  illuminate  the  object  under  inspection,  and  a  video  camera  for  taking  images  of 
the  object.  Conceptually,  the  problem  we  are  solving  has  three  parts;  shape  extraction,  pixel 
segmentation,  and  roughness  extraction.  The  shap>e  information  is  produced  directly  by 
three  light  and  four  light  photometric  stereo  methods.  After  we  have  shape  information,  we 
can  apply  statistical  segmentation  techniques  to  determine  which  pixels  are  specular  and 
which  are  nonspecular.  Then,  we  can  use  the  specular  pixels  and  shape  information,  in  con¬ 
jugation  with  the  simplified  Torrance-Sparrow  reflectance  model  to  determine  the  surface 
roughness.  The  method  has  successfully  been  applied  to  a  number  of  synthetic  and  real 
objects. 
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1.  Introduction 

In  modem  manufacmring  environments,  part  inspection  is  an  important  part  of  quality  con¬ 
trol.  Today,  the  majority  of  inspection  tasks  are  performed  manually.  Manual  inspection  is 
subject  to  human  error,  is  monotonous,  and  is  very  labor  intensive.  In  an  effort  to  automate 
part  inspection,  some  companies  have  turned  to  computer  vision  techniques.  However,  these 
efforts  have  predominately  been  limited  to  two  dimensional  measurement.  Two  dimensional 
measurement,  while  not  easy,  is  much  simpler  than  three  dimensional  measurement.  This  is 
because  conventional  cameras  see  in  two  dimensions.  Complex  algorithms  are  required  to 
convert  a  camera’s  two  dimensional  view  into  three  dimensions.  However,  the  ability  to 
measure  parts  in  three  dimensions  would  be  an  important  tool  for  inspecting  manufactured 
parts.  Two  basic  three  dimensional  measurements  that  are  made  on  many  manufactured 
parts  are  the  measurement  of  the  shape  and  surface  roughness  of  the  part. 

Computer  vision  research  has  produced  a  number  of  basic  techniques  for  measuring  the  sur¬ 
face  shape  of  an  object:  stereo  vision,  range  finders,  and  photometric  techniques.  Photomet¬ 
ric  techniques  use  image  intensity  to  determine  shap)e.  There  are  two  basic  photometric 
techniques  for  transforming  from  image  intensity  to  shap>e:  exp>erimentally  derived  reflec¬ 
tance  map  techniques  and  physically  based  theoretical  reflectance  models. 

A  reflectance  map  contains  a  transformation  between  surface  orientation  and  image  bright¬ 
ness  for  a  given  material,  light  source  (direction  and  intensity),  and  viewing  direction.  Silver 
[1]  described  how  to  experimentally  determine  a  reflectance  map.  In  order  to  construct  a 
reflectance  map,  a  sphere  needs  to  be  constructed  of  the  material  to  be  inspected.  Then,  an 
image  of  the  sphere,  which  contains  all  surface  orientations,  is  taken  with  the  specified  light¬ 
ing  and  viewing  geometries.  The  reflectance  map  is  constructed  from  this  image.  Exp>eri- 
mentally  derived  reflectance  maps  are  basically  lookup  tables  between  image  intensity  and 
surface  orientation.  If  the  material,  lighting,  or  viewing  geometries  changes,  a  new  calibrat¬ 
ing  image  needs  to  be  taken,  and  a  new  reflectance  map  has  to  be  calculated. 

Physically  based  reflectance  models  [2]  express  the  relationship  between  image  brighmess, 
imaging  geometry,  lighting  geometry,  and  surface  shajje  in  the  form  of  an  explicit  liijthe- 
matical  function.  Since  the  parameters  of  the  reflectance  model  are  expressed  as  a  function, 
imaging  and  lighting  geometries  can  be  changed  without  the  recalibration  reo’  i^ed  by  the 
experimentally  derived  reflectance  map  technique.  There  are  three  basic  categories  of 
reflectance  models,  based  on  surface  type.  Diffuse  surfaces  follow  the  lanibertian  reflec¬ 
tance  model.  The  intensity  of  a  lambertian  surface  is  only  dejjendent  on  the  angle  between 
il'.e  light  source  direction  and  the  surface  orientation.  Shiny  surfaces  follow  the  specular 
spike  reflectance  model.  For  specular  spike  surfaces,  a  strong  refl*^  ction  is  observed  when 
the  angle  of  incidence  equals  the  angle  of  reflectance.  Surfaces  that  exhibit  a  specular  lobe 
(A  specular  lobe  exhibits  a  gaussian  intensity  profile,  around  its  sptecular  peak.)  on  top  of  a 
lambertian  base  follow  the  Torrance-Sparrow  reflectance  model[3].  The  Torrance-Spanow 
model  allows  surface  roughness  to  be  determined  in  addition  to  surface  shap>e. 

We  seek  to  develop  a  method  that  can  extract  the  srtface  shajje  and  surface  roughness  of  an 
object  that  exhibits  a  specular  lobe.  The  method  which  we  call  “four  light  photometric  ste- 
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reo”,  uses  four  lights  which  sequentially  illuminate  the  object  under  inspection,  and  a  video 
camera  for  taking  images  of  the  object. 

Conceptually,  the  problem  we  are  solving  has  three  parts:  shape  extraction,  pixel  segmenta¬ 
tion,  and  roughness  extraction.  The  shape  information  is  produced  directly  by  the  three  light 
and  four  hght  photometric  stereo  methods.  After  we  have  shap>e  information,  we  can  apply 
different  techniques  to  determine  which  pixels  are  specular  and  which  are  nonspecular. 
Then,  we  can  use  the  specular  pixels  and  shape  information,  in  conjugation  with  the  simpli¬ 
fied  Torrance-Sparrow  reflectance  model  to  determine  the  surface  roughness. 

In  section  two  we  develop  the  four  light  photometric  stereo  method  for  different  regions  of  a 
gaussian  sphere.  In  section  three,  we  develop  a  simplified  version  of  the  Torrance-Sparrow 
reflection  model,  and  we  develop  the  roughness  extraction  algorithm.  Section  four  discusses 
various  implementation  issues.  Results  are  presented  in  section  five.  In  Appendix  A,  we 
review  the  three  hght  photometric  stereo  method  developed  by  Woodham. 

1.1.  Previous  Work 

Woodham  [4]  proposed  the  photometric  stereo  method  to  determine  the  surface  shape  of 
lambertian  (diffuse)  dominant  surfaces  by  using  three  point  light  sources,  and  a  reflectance 
map  for  each  light  source. 

A  number  of  methods  have  been  developed  to  recover  the  shape  of  specular  spike  objects. 
Ikeuchi  [5]  used  three  extended  sources  to  determine  the  surface  orientation  of  specular  sur¬ 
faces.  He  determined  the  radiance  distribution  of  the  extended  sources,  and  used  this  to  pre¬ 
dict  image  intensity  based  on  surface  orientation.  Nayar,  Weiss,  Simon,  and  Sanderson  [6] 
developed  a  system  that  used  127  point  sources  to  determine  the  shape  of  specular  objects. 
Coleman  and  Jain  [7]  proposed  using  four  lights  to  detect  specularities.  They  assumed  that 
light  source  specularities  were  non-overlapping,  and  that  if  one  light  was  specular,  a  valid 
surface  orientation  could  be  determined  with  the  remaining  three  lights. 

Healey  and  Binford  [8]  used  a  simplified  version  of  the  Torrance-Sparrow  reflectance  model 
to  recover  object  curvature  around  the  specular  peak  of  specular  lobe  objects.  They  assumed 
that  the  surface  roughness  of  the  object  was  known.  Nayar,  Ikeuchi,  and  Kanade  [9]  devel¬ 
oped  a  method  for  surfaces  that  exhibited  a  lambertian  and  specular  spike  component.  The 
method  used  an  array  of  extended  light  sources  to  illuminate  an  object  from  different  direc¬ 
tions.  The  array  of  extended  sources  guaranteed  that  specular  reflections  were  detected.  In 
addition,  they  were  able  to  determine  the  ratio  of  the  specular  to  lambertian  reflection,  which 
is  related  to  surface  roughness.  Ikeuchi  and  Sato  [10]  developed  a  method  to  recover  the 
shape  and  roughness  of  objects  that  follow  the  Torrance-Sparrow  reflectance  model.  They 
took  a  range  image  using  a  range  finder,  and  an  intensity  image  using  a  CCD  camera,  using 
the  range  image  to  determine  surface  shape.  Then,  they  used  an  iterative  segmentation  algo¬ 
rithm  to  classify  pixels  in  the  intensity  image  as  specular  or  lambertian,  and  to  determine  the 
light  source  direction.  Surface  roughness  was  determined  by  fitting  specular  intensity  values 
to  a  simplified  version  of  the  Torrance-Spanow  reflectance  model.  The  use  of  a  range  finder 
to  determine  surface  orientation  requires  taking  first  derivatives  of  the  range  data.  This 
makes  the  method  sensitive  to  noise. 
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2.  Determining  Surface  Shape  and  Pixel  Segmentation 
using  Four  Light  Photometric  Stereo 


An  object  illuminated  by  four  light  sources  will  produce  three  categories  of  regions  based 
upon  illumination:  regions  illuminated  by  all  four  light  sources,  regions  illuminated  by  three 
light  sources,  and  regions  illuminated  by  only  two  light  sources.  If  we  Uluminate  an  object 
with  four  equally  spaced  light  sources,  the  region  map,  represented  on  the  gaussian  sphere, 
will  look  like  this: 


Gaussian  Sphere 


Light  4 


Region  1  is  illuminated  by  all  four  light  sources.  Regions  2,  3,  4,  and  5  are  illuminated  by 
three  light  sources.  Regions  6,  7,  8,  and  9  are  illuminated  by  two  light  sources.  The  region 
boundaries  are  formed  by  the  shadow  lines  of  each  light  source.  The  size  and  shape  of  each 
region  is  dependent  on  the  inclination  of  the  light  sources. 

Different  information  is  available  in  each  of  the  three  categories  of  regions.  Therefore,  we 
use  different  techniques  for  determining  surface  shape  and  for  performing  pixel  segmenta¬ 
tion  in  each  region. 

2.1.  Determining  Surface  Shape  and  Pixel  Segmentation  in  the  Four  Light 
Illuminated  Region 

Coleman  and  Jain  proposed  using  four  lights  to  determine  the  shape  of  surfaces  that  were 
nonlambertian.  The  method  is  only  valid  in  regions  illuminated  by  all  four  light  sources,  the 
region  labeled  1 .  They  proposed  to  calculate  four  albedo  values  based  on  the  four  possible 
combinations  of  three  light  sources.  For  a  perfectly  lambertian  surface,  the  four  albedos 
would  be  identical.  But,  for  surfaces  that  exhibit  some  specularity,  this  is  not  the  case.  If  we 
assume  that  the  specular  lobes  of  each  light  source  do  not  intersect,  then  a  sp>ecularity  in  one 
light  source  will  cause  the  three  albedos  that  use  that  light  source  to  be  high,  while  the 
albedo  that  does  not  use  the  light  source  will  be  low.  Given  four  intensity  values,  (II,  12, 13, 
14),  and  four  light  source  directions,  (SI,  S2,  S3,  S4),  we  can  define  four  albedos,  (Ra,  Rb,  Rc, 


Rd),  as  follows: 


Six  Sly  51r 

Six  Sly  Slz 

'S3x  S3y  S3z 

rS4.r  S4y  S4r 

Sa  = 

Six  Sly  Slz 

Sb  = 

S3x  S3\  S3z 

Sc  = 

S4x  S4y  S4z 

Sd  =  Six  Sly  Slz 

S3x  S3y  S3z 

_S4x  S4y  S4z 

Six  Sly  Sir 

Ls2;r  S2y  S2z 

71 

1/2 

74 

la  = 

11 

Ib  =  1/3 

Ic  = 

74 

Id  = 

71 

L/3 

L/4_ 

71 

11 

Ra  -  1  (Sa)  'la| 

Rb  =  i(Sb)”'lb| 

Rc  =  |(Sc)“'lc| 

Rd  =  l(Sd)''ld| 

(Six,  Sly,  and  Slz  are  the  x,  y,  and  z  components  of  the  unit  vector  to  light  source  number 
one) 

If  II  is  specular,  Ra,  Rc,  and  Rd  will  be  elevated  above  their  lambertian  levels.  Rb  wUl  be 
the  lambertian  albedo,  since  12, 13,  and  14  are  not  specular  (Our  assumption  is  that  for  each 
image  point,  at  most  one  light  source  will  be  specular.). 

Therefore,  we  can  identify  the  nonspecular  light  sources,  by  using  the  four  albedos,  and  we 
can  use  these  nonspecular  light  sources  to  produce  a  valid  surface  normal.  In  the  above 
example,  we  would  determine  that  Rb  is  the  minimum  albedo,  and  then  use  12, 13,  and  14  to 
determine  the  surface  normal.  Since  the  surface  is  lambertian,  the  surface  normal  is  equal 
(see  Appendix  A)  to: 

r  n-'r  -1  -  n 

Six  Sly  Slz  II  Nx 

S3x  S3y  S3z  13  =  Ny 

S4x  S4y  S4^  [mJ  Nz_ 

However,  due  to  image  noise,  the  four  albedos  will  never  be  exactly  equal.  We  need  to 
establish  a  threshold  to  determine  when  the  differences  in  albedo  indicate  a  specularity,  and 
when  they  are  just  due  to  random  events.  Coleman  and  Jain  define  albedo  deviation,  Rdev: 

Rdev  =  (  ^  {Ri  -  Rmean))  /  {^Rmin) 

I  =  a,  b,  c,  d 

If  Rdev  >  Rt,  they  classify  the  pixel  as  specular,  otherwise  it  is  lambertian.  Rt  is  a  manually 
selected  threshold.  The  use  of  an  arbitrarily  selected  threshold  makes  this  classification 
scheme  ad  hoc. 

We  propose  a  better  classification  scheme  which  would  use  the  variance  of  the  camera’s 
intensity  response  to  determine  a  statistically  meaningful  threshold.  If  Oj*  is  the  camera’s 
intensity  variance  measured  at  a  particular  pixel,  then  we  can  establish  a  specular  threshold 
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based  on  a  +/-3a  distribution  [11],  We  assume  that  the  light  source  directions  are  known 
without  any  uncertainty.  Then,  in  the  case  where  R1  is  the  maximum  albedo  and  R2  is  the 
minimum  albedo,  we  can  define  Rdev  as: 

Rdev  =  Rl -R2 


If  Rdev  is  greater  than  the  specular  threshold,  one  of  the  light  sources  is  specular. 


We  can  determine  an  accurate  surface  normal  if  a  pixel  is  specular,  by  using  the  light 
sources  from  the  minimum  albedo  to  determine  the  surface  normal.  If  a  pixel  is  not  specular, 
we  can  use  any  of  the  three  light  sources  to  determine  the  surface  normal. 


2.1.1.  Specular  Threshold  in  Four  Light  Illuminated  Region 


A  pixel  is  specular  if  Rdev  is  greater  than  the  specular  threshold: 


If  an  albedo  does  not  use  a  particular  light  source,  the  terms  involving  its  partial  derivatives 
simply  go  to  zero.  The  partial  derivatives  are  not  complex,  and  exact  expressions  can  easily 
be  derived.  This  threshold  assumes  a  worst  case  6a  separation  between  Rl  and  R2.  In  prac¬ 
tice  each  pixel  in  the  image  has  its  own  light  intensity  variance,  due  to  variations  in  manu¬ 
facture  of  the  camera’s  CCD  array.  Therefore,  the  variance  needs  to  be  measured  for  each 
pixel. 

2.2.  Determining  Surface  Shape  and  Pixel  Segmentation  in  the  Three 
Light  Illuminated  Region 

The  region  labeled  3  is  illuminated  by  light  sources  2,  3,  and  4.  Its  borders  are  formed  by  the 
shadow  lines  of  light  sources  1 , 4,  and  2.  If  we  assume  that  the  specular  regions  of  each  light 
source  are  nonoverlapping,  and  if  the  surface  is  specular,  region  3  may  have  a  specularity 
from  light  source  3.  Illumination  by  light  sources  2  and  4  will  be  nonspecular  in  region  3. 
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Under  these  assumptions,  we  can  determine  an  accurate  surface  normal  using  light  source  2, 
light  source  4,  and  the  shadow  line  of  light  source  1.  We  do  not  use  light  source  3  because  it 
may  be  specular.  We  can  write  three  equations,  in  three  unknowns: 

U 

-  =  S2xNx^S'lyhl\-\-SlzNz 

P  '  ' 

-  =  SAxNx  +  54yA^>'  +  SAzNz 
P 

{a/.t)^+  (/vy)"+  =  1 

The  albedo,  p,  the  light  source  directions  S2  and  S4,  and  the  image  intensities  12  and  14  are 
known.  The  albedo  is  calculated  form  the  albedos  of  the  lambertian  pixels  within  a  10X10 
pixel  area  within  the  four  light  illuminated  region.  The  surface  normal  is  by  definition  a  unit 
vector. 


This  set  of  three  equations  has  two  sets  of  solutions.  If  we  define  the  following  intermediate 
variables: 


Six  _  Sly  _  72 

S2z  ~  Slz  ^  ~  pSlz 


h  = 


74 

p54r 


-g  +  h 
c-a 


/  = 


d~b 


c-a 


A  =  fa-  +/  -  labf+  b'  +  1 
B  =  2agf-2efa~ -2ef+2abe  -  2bg 
C  -  -  2age  +  a  e"  +  e"  +  -  1 


The  two  sets  of  surface  normal  solutions  are: 

-B  +  Jb-  -  4AC 
=  — 2A — 

Nx^  =  e-fNy^ 

Nz^  =  g-  aNx^  -  bNy^ 


Ny^  = 


-B-Jb--  aac 

Nx-,  =  e  -fNy^ 

Nz2  =  g  -  aNx-,  —  bNy2 


The  proper  solution  can  be  selected  by  using  the  boundary  condition  of  light  source  one’s 
shadow  line.  (It  is  possible  that  due  to  degenerate  positioning  of  light  sources  1,  2,  and  4, 
that  both  surface  normals  are  on  the  same  side  of  light  source  one’s  shadow  line.  We  do  not 
consider  this  case.)  On  the  shadow  line  of  light  source  one,  the  following  equation  holds: 

SlxNx  +  SlyNy  +  SlzNz  =  0 


In  region  number  3, 


Slxb/x  +  5 1  vNy  +  5 1  rA/;  <  0 


So  we  can  select  the  proper  solution  by  seeing  if  this  condition  holds  for  each  solution. 
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2.2.1.  Threshold  for  Selecting  Proper  Surface  Normal  Solution 


Selecting  the  proper  solution  becomes  more  complex  in  the  presence  of  intensity  noise, 
which  causes  uncertainty  in  the  normals,  and  uncertainty  in  the  location  of  the  shadow  line. 
We  assume  that  the  light  source  directions  are  known  without  any  uncertainty.  The  intensity 
values  and  the  albedo  have  variances  (The  albedo  variance  is  calculated  from  the  albedos  of 
the  lambertian  pixels  within  a  10X10  pixel  area  within  the  four  light  illuminated  region.). 
Based  on  these  variances,  we  can  express  the  uncertainty  of  the  surface  normals.  (The  par¬ 
tial  derivatives  of  these  expressions  are  very  complex.  Therefore,  the  partial  derivatives  are 
approximated  with  a  discrete  partial  derivative.)  We  will  continue  using  the  example  above. 
The  light  source  variances  are  crji  and  cri4.  The  albedo  variance  is  av  The  uncertainty  of  the 
surface  normals  is: 


dp 


i) 


+ 


O  A7- 


A  similar  set  of  expressions  can  be  derived  for  the  variances  of  the  second  set  of  surface  nor¬ 
mals.  Using  this  set  of  variances  we  can  decide  whether  a  pixel  is  on  the  correct  side  of  the 
shadow  line.  For  a  pixel  to  confidently  be  in  region  niunber  3,  the  following  relation  must 
hold  for  its  surface  normal: 

SlxNx^+SlyNy^+SlzN:^<3JG-Nx^(Slx)'  +  crsy^(S\y)-  +  a-s:^{Slz)- 


This  expression  expresses  with  a  3a  confidence  that  the  surface  normal  solution  is  in  the 
three  light  illuminated  region,  region  number  3.  We  will  have  a  similar  expression  for  Nx-,, 
Ny2,  Nzi.  It  is  possible  that  we  cannot  confidently  say  that  either  set  of  surface  normals  is  in 
region  number  3.  (This  expression  is  false  for  both  sets  of  surface  normals.)  These  pixels  are 
flagged  with  a  special  classification. 


At  this  point,  we  have  derived  how  to  confidently  determine  surface  normals  for  the  three 
light  Uluminated  region.  We  also  need  to  classify  pixels  as  specular  or  lamberti  m. 

2.2.2.  Specular  classification  of  Pixels  in  the  Three  Light  Illuminated  Region 

In  order  to  classify  a  pixel  as  specular,  we  can  determine  whether  its  measured  brighmess, 
with  respect  to  a  given  light  source,  is  larger  than  its  predicted  lambertian  brightness  for  the 
same  light  source.  We  can  determine  the  predicted  lambertian  brighmess  of  a  pixel  -n  the 
three  light  illuminated  region  by  using  the  surface  normal  derived  in  the  preceding  section, 
the  source  dire^aons  (which  are  known),  and  the  lambertian  albedo.  Using  the  example  of 
the  preceding  section,  we  want  to  determine  if  the  measured  brightness  due  to  light  source 
three  is  greater  than  its  lambertian  brighmess.  If  the  derived  nonnal  vector  for  the  pixel 
under  consideration  is  (Nx,  Ny,  Nz),  and  the  measured  brightness  of  the  pixel  due  to  light 
source  three  is  13,  then  the  pixel  is  .specular  if: 
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73  >  p  (SSxNx  -t-  S3yNy  +  S3zN:) 

2^.2.1.  Specular  Threshold  in  the  hree  Light  niuminated  Region 

We  need  to  consider  the  uncertainty  of  13,  p,  Nx,  Ny,  and  Nz.  The  uncertainty  of  the  pre¬ 
dicted  lambertian  brightness,  Earn,  is  o'liam- 


/iam  =  p  (S2xNx  +  SSyf^y  +  53:N:) 


Nz 


If  o'o  is  the  uncertainty  of  the  measured  brightness,  then  we  can  say  that  the  pixel  is  sp 
lar,  assuming  a  3a  distribution,  if: 


-U- 


r~^  ^ 

2.3.  Determining  Surface  Shape  and  Pixel  Segmentation  in  the  Two  Light 
Illuminated  Region 

It  is  possible  to  determine  the  surface  normal  in  the  region  iEuminated  by  only  two  lights. 
The  region  labeled  7  is  illuminated  by  light  sources  2  and  3.  Its  borders  are  formed  by  the 
shadow  lines  of  light  sources  1  and  4. 


If  we  assume  that  neither  light  2  nor  ng/it  3  have  a  specularity  in  region  7,  we  can  write  the 
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following  set  of  equations: 

-  =  S2xNx  +  S2vN\  +  S2zN: 
P 


-  =  S3>xNx  +  S3>\N\-  +  S2:Nz 
P 

iNx)-+  (Ny)-+  (Nz)~  =  1 

The  equations  will  yield  two  sets  of  surface  normal  solutions,  and  can  be  solved  using  the 
methods  of  the  three  light  illuminated  region. 

The  proper  solution  can  be  selected  by  using  the  boundary  condition  for  light  source  one ’s 
shadow  line,  and  the  boundary  condition  for  light  source  four’s  shadow  line.  In  region  num¬ 
ber  7, 

SlxNx  +  S 1  vA/y  +  S 1  r/Vr  <  0  a  S4xNx  +  S4yNy  +  S4zNz  <  0 

2.3.1.  Threshold  for  Selecting  Proper  Surface  Normal  Solution 

If  we  consider  the  uncertainty  due  to  intensity  noise,  the  following  two  relations  must  hold: 

S'IaA/a'i  +  51yA/yj  +Slz^'z^  <  3  Ja'yv’x,  (51x)"-t-  (51y)"  +  (Sir) * 

S4A:A^jir,  +S4yNy^  +S4zWz^  <  3  ‘  +  (•S^y) ‘  +  (S4r)‘ 

where  (Nx|,NyiNzi)  is  one  of  the  surface  normal  solutions.  A  similar  set  of  equations  hold 
for  the  second  set  of  surface  normal  solutions.  The  surface  normal  vanances  are  calculated 
using  the  expressions  derived  for  the  three  light  illuminated  region. 

2.3.2.  Specular  Classification  of  Pixels  in  the  Two  Light  Illuminated  Region 

The  assumption  that  light  source  one  and  four  do  not  produce  a  specularity  in  the  two  light 
illuminated  region  is  not  strong.  However,  if  one  of  the  lights  does  produce  a  specularity,  the 
surface  normal  solutions  may  become  imaginary.  If  the  lambertian  albedo  is  correct,  and 
there  are  no  interreflections,  then  the  existence  of  imaginary  surface  normal  solutions  indi¬ 
cates  a  specularity  in  one  of  the  two  light  sources.  Pixels  may  not  be  sufficiently  specular  to 
cause  the  surface  normal  solutions  to  become  imaginary.  In  this  case,  the  surface  normal 
solutions  will  be  erroneous,  and  it  will  be  not  be  possible  to  detect  the  condition. 
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3.  Extracting  Specular  Intensity  and  Surface  Roughness 

The  Torrance-Sparrow  reflectance  model  allows  us  to  determine  the  specular  intensity  and 
surface  roughness  of  an  object.  In  this  section,  we  develop  a  simplified  version  of  the  Tor¬ 
rance-Sparrow  model.  Then,  we  develop  the  specular  intensity  and  surface  roughness 
extraction  algorithms. 

3.1.  Simplified  Torrance-Sparrow  Model 

The  Torrance-Sparrow  model  is  a  geometrical  optics  model  of  reflection  for  rough  surfaces. 
The  model  describes  reflection  for  surfaces  that  exhibit  a  specular  lobe. 


Geometry  for  Torrance-Sparrow  Model 


For  a  fixed  incident  light  angle  (which  we  assume  because  the  light  is  far  from  the  object, 
making  the  light  rays  parallel  over  the  object’s  surface),  and  for  a  given  material,  the  surface 
radiance  is: 


E  =  f  (e.',n') 


rG(e,.e^.V 

cos  9 


exp{-c~cC) 


F(e,',n  )  is  the  Fresnel  reflectance  coefficient.  (9,'  is  the  local  angle  of  incidence,  n  'is  the  local 
surface  orientation.  9;  is  the  global  angle  of  incidence,  n  is  the  mean  surface  orientation.)  For 
insulators,  with  incident  angles  of  less  than  40  degrees,  the  Fresnel  coefficient  can  be 
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approximated  by  a  constant. 


Fresnel  Reflectance  -  Magnesium  Oxide 

F(e>) 


G(0,,0p(!)p)  is  the  geometric  attenuation  factor.  This  accounts  for  masking  and  shadowing  of 
one  micro  facet  by  adjacent  micro  facets.  For  incident  angles  between  0  degrees  and  60 
degrees,  and  reflected  angles  between  0  degrees  and  80  degrees,  the  geometric  attenuation 
factor  is  close  to  unity. 


Typical  Plot  of  Geometric  Attenuation  Factor 


With  these  assumptions  we  can  simplify  the  Torrance-Sparrow  model  to: 

'y  -) 

'  exp{-c~cC) 


E  =  B 


COS0 


B  is  a  constant  which  we  will  call  the  specular  intensity. 

If  the  camera  is  at  N=(0,0,1),  then  cosQ^  simplifies  to  Nz  (the  z  component  of  the  surface  nor¬ 
mal).  “c”  is  a  constant  that  is  proportional  to  the  surface  roughness.  So  c*  can  be  replaced 
with  “K”.  If  we  determine  K,  we  can  find  out  how  rough  a  surface  is.  We  then  have; 


The  total  surface  radiance  will  be  the  sum  of  the  underlying  lambertian  radiance.  A,  and  the 
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radiance  of  the  specular  lobe. 


E  =  A+B 


exp  {-Ka~) 


3.2.  Specular  Intensity  and  Surface  Roughness  Extraction  Algorithm 

Once  the  surface  shape  is  determined  and  the  specular  pixels  are  extracted,  the  specular 
intensity  and  surface  roughness  can  be  detemtined.  In  the  simplified  Torrance-Sparrow 
model,  we  need  to  determine  B,  the  specular  intensity,  and  K,  the  specular  sharpness  which 
is  proportional  to  the  surface  roughness.  We  use  the  follow'ing  algorithm: 

1 .  Using  the  extracted  surface  normals  and  lambertian  albedo,  we  can  deter¬ 
mine  the  lambertian  intensity  at  each  image  point. 

>1  =  p  ( SxNx  +  SyNy  +  SzN:) 

2.  We  can  define  D,  the  difference  between  the  measured  image  brightness 
at  each  specular  pixel  and  the  lambertian  intensity. 

D  =  I-A 


3.  CX  can  be  determined  from  the  light  source  direction,  which  is  known,  and 
from  the  surface  normal,  which  is  also  known. 

4.  B  and  K  can  be  determined  by  an  iterative  least  square  fitting.  If  we  start 
out  with  an  initial  guess  for  B,  we  can  estimate  K. 

Our  model  is; 

We  can  linearize  the  model  with  respect  to  K,  by  taking  the  natural  log; 
InD  +  InNz  -  InB  +  KkT  =  0 

The  difference  between  the  D  and  the  specular  part  of  the  Torrance-Spar¬ 
row  model,  using  our  current  estimate  of  B  and  K,  is  the  error  in  fit 
between  our  measurements  and  our  model: 

InD  +  InNz  —  InB  +  KcC  =  e. 

The  total  error  over  all  the  speculcU"  pixels,  (i.j),  in  the  image  is: 

*> 

P  =  "^[InD  (i.j)  +  InNz  (i.j)  ~  InB  +  Ka~  (i.j)] 

We  want  to  find  the  minimum  of  the  error  with  respect  to  K.  This  happens 
when: 

^  -  0  =  (InDii.j))  +  ((,;)( /n/Vr  ( f,;) )  -  InB'^a.'  (i.j) 

I.J  I.J  I.J  I.J 
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K  is  then  simply; 

-^a“  (i,;")  lnD(i,j)  -  ^a"  {i,j)  InNz  (i.j)  +  InB'^aT  (i,j) 

_ w _ U. _ 

'ZaiiJ) 

‘■j 

5.  Using  the  value  of  K,  just  obtained,  we  can  derive  a  value  for  B. 


The  expression  for  the  difference  between  D  and  the  specular  part  of  the 
Torrance -Sparrow  model  is  already  linear  in  B.  We  can  write  an  expres¬ 
sion  for  the  error  over  all  the  specular  pixels  as; 

->  n- 

Bexpi-Ka"  U-j)) 

Nr  (/.;•)  _ 


r 

P  =  X\D{iJ) 
‘•i  ^ 


Taking  the  partial  derivative  of  the  total  error  with  respect  to  B,  we 
obtain; 


dP  (exp{-Ka-(i,j))D{iJ)) 

=  “  =  5 - imw) - 


dB 

Therefore  B  is; 


(exp(-Kar  (i,j))) 
(Nz(uj)) 


I 

B  =  id 


expi-Ka~(iJ))D(i,j) 

NzUJ) 


[  {exp{-Ka‘(i,J)))l 
f-L  (Nr  (/,;•))  j 


6.  Steps  4  and  5  are  repeated  until  the  values  of  B  and  K  converge. 
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4.  Implementation 

4.1.  Determining  Light  Source  Directions 

The  accuracy  of  the  light  source  directions  is  very  imponant  to  the  determination  of  the  sur¬ 
face  shape  and  consequently  to  the  extraction  of  the  surface  roughness.  A  number  of  tech¬ 
niques  have  been  tested.  The  simplest  technique  is  to  use  the  brightest  point  on  the  surface 
of  a  lambertian  sphere.  The  normal  of  the  brightest  point  is  coincident  with  the  vector  point¬ 
ing  to  the  light  source.  Although  the  method  is  simple  to  implement,  it  is  sensitive  to  noise. 
Since  the  single  brightest  point,  or  a  small  group  of  the  brightest  points,  is  used  to  determine 
the  source  direction,  image  intensity'  noise  makes  their  absolute  magnitude  uncertain. 

A  second  method  is  to  fit  a  lambertian  reflectance  function  (a  cosine)  to  the  100  brightest 
pixels.  Since  the  data  around  the  peak  of  the  cosine  is  flat,  the  fit  is  not  very  stable.  If  the 
number  of  points  (the  threshold  used  to  select  the  points)  changes,  the  light  source  direction 
will  change.  This  is  not  very'  reassuring. 

This  method  can  be  significantly  improved  by  obtaining  a  more  stable  fit.  A  more  stable  fit 
can  be  obtained  by  using  points  extending  from  the  brightest  part  of  the  cosine,  to  the 
shadow  area  of  the  cosine.  This  fit  will  be  stable.  If  the  threshold  is  changed,  within  reason¬ 
able  amounts,  the  light  source  direction  will  not  change. 

We  find  the  light  source  directions  by  performing  a  least  square  fit  on  the  intensity  data  of  a 
lambertian  sphere.  The  least  squares  formulation  contains  a  residual  term.  This  term  allow's 
the  dark  current  value  of  the  camera  to  be  recovered.  This  value  is  subsequently  subtracted 
from  all  images. 

The  intensity  at  each  point  on  the  lambertian  sphere  is: 

A  =  p  ( SxNx  ( i,  j)  -I-  SyNy  ( /,  j)  +  SzNz  ( i.  j) ) 


The  total  error  over  the  entire  image  between  the  measured  intensity,  E(i,j),  and  the  intensity 
given  by  the  lambertian  model,  using  our  solution  for  (Sx,Sy',Sz,D),  where  D  is  the  residual, 
dark  current  term,  is: 

P  =  ^[E{i.j)  -  SxNx  (i.j)  -  SyNy  {i.j)  -SzNz{i,j)  -  D]~ 


We  want  to  find  the  minimum  of  the  error  with  respect  to  (Sx,Sy,Sz,D).  This  happens  when: 


dP 

dSx 


=  0 


'^Nxii.j)  (E(i.j)  ~SxNx(uj)  -SyNy(i.j)  -SzNzd.j)  -  D) 


^  =  0  =  (/,;■)  (£(/,;■)  -SxNx(i,j)  -SyNy(i.j)  -SzNz(i.j)  -  D) 

^  =  0  =  TNzii.j)  {EUJ)  -SxNxii.j)  -SvNx(i.j)  -SzNz{i,j)  -  D) 
^  =  0  =  ^E(i.j)  -  SxNx  (i.j)  -  SyNy  (i.j)  -SzNzd.J)  -D 
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Tliis  set  of  four  equations  can  be  solved  directly  for  Sx,  Sy,  Sz,  and  D. 

4.2.  Image  Intensity  Normalization 

Normalization  of  the  four  image  intensities,  to  compensate  for  differences  in  illumination 
strength,  is  critical  to  extracting  surface  shape,  and  subsequently  to  the  extraction  of  surface 
roughness.  The  most  common  technique  uses  the  brightest  pixel  in  the  image  as  a  normaliz¬ 
ing  factor,  and  is  susceptible  to  image  intensity  noise  inaccuracies.  Using  a  larger  number  of 
points  is  makes  this  technique  more  robust. 

In  order  to  normalize  the  light  sources,  each  light  source  needs  to  illuminate  points  that  have 
the  same  camera/surface  normal/light  source  geometry.  Theoretically  these  points  will  have 
the  same  brightness.  An  arbitrary  object  is  not  guaranteed  to  have  such  a  set  of  points.  Our 
solution  is  to  use  the  lambertian  sphere.  The  sphere  exposes  the  same  set  of  surface  normals 
to  aU  light  sources. 

After  we  have  solved  for  the  light  source’s  direction,  we  can  fit  a  cosine  to  a  plot  of  intensity 
versus  incident  angle.  The  magnitude  of  the  cosine  is  our  normalization  factor. 

Below  is  a  image  of  a  lambertian  sphere,  a  plot  of  intensity  versus  incident  angle  (using  the 
extracted  light  source  direction)  and  a  best  fit  cosine  (magnitude  =  143). 
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4.3.  Image  Linearization 

Based  on  the  known  optical  densities  of  the  CMU  Color  Chart,  and  the  calculated  illumina¬ 
tion  strength,  expected  intensities  from  our  CCD  camera  can  be  computed.  An  image  is 
taken  of  the  calibration  chart.  TTie  measured  and  computed  intensities  are  compared,  and  a 
table  of  correction  factors  is  generated.  Then,  images  taken  with  the  camera  can  be  linear¬ 
ized  by  using  the  table. 

4.4.  Image  Intensity  Variance 

The  image  intensity  variance,  tTj,  was  determined  by  taking  100  intensity  images  of  a  fixed 
scene,  with  the  CCD  camera.  The  intensity  variance  was  then  computed,  for  each  pixel, 
from  the  100  intensity  values. 


4.5.  Imaginary  Surface  Normal  Solutions  in  Three  Light  Region 


It  is  possible  that  the  surface  normal  solutions  in  the  three  light  illuminated  region  will  be 
imaginary.  This  can  happen  when  one  pixel  is  brighter  than  a  lambertian  pixel  could  be,  as 
interreflection  or  an  unexpected  specularity,  would  cause.  It  can  also  occur  if  there  is  an 
error  in  the  lambertian  albedo,  p.  The  existence  of  an  imaginary  solution  can  be  understood 
with  the  foUowing  description.  The  solution  to  the  set  of  equations  corresponds  to  finding 
the  points  on  a  lambertian  gaussian  sphere  that  are  the  intersections  of  two  isobright  con¬ 
tours.  One  isobright  contour  is  caused  by  S2,  and  has  intensity  12.  The  other  isobright  con¬ 
tour  is  produced  by  S4,  and  has  intensity  14.  If  we  imagine  that  we  are  illuminating  a 
gaussian  sphere  (which  contains  all  surface  normals),  these  two  isobright  contours  will  be 
two  circles.  The  two  circles  wUl  in  general  intersect  in  at  most  two  places.  These  two  inter¬ 
section  points  correspond  to  the  two  real  solutions  to  the  surface  normal  equations.  How¬ 
ever,  if  one  circle  gets  smaller  (which  corresponds  to  the  pixel  getting  brighter,  as  might  be 
caused  by  interreflection),  at  some  point  the  isobright  contours  wiQ  fail  to  intersect.  When 
the  contours  fzul  to  intersect,  the  solutions  to  the  surface  normal  equations  will  become 
imaginary. 
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4.6.  Consistent  Segmentation 

There  is  no  guarantee  that  the  statistical  segmentation  of  specular  and  lambertian  pixels 
within  the  three  light  illuminated  region  and  the  four  Light  illuminated  region  w'ill  produce 
consistent  segmentations.  The  uncertainty  of  the  segmentation  in  the  three  light  illuminated 
region  has  a  higher  uncertainty  than  the  segmentation  in  the  four  light  Uluminated  region. 
The  derivation  of  the  surface  nonnals  in  the  three  light  illuminated  region  involves  a  lot  of 
computation;  this  increases  their  variance,  making  cr[|am  large. 

In  order  to  produce  consistent  segmentations,  w'e  increase  the  uncertainty  threshold  of  the 
four  light  illuminated  region  by  2X,  to  match  three  light  illuminated  region.  The  amount  of 
increase  was  found  by  observing  the  segmentation  of  synthesized  images. 


Therefore,  a  pixel  in  the  four  light  illuminated  region  is  specular  if: 


Rdev  >  1 


:  i(^R\  + 

\  d/1  d/1 
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4.7.  Extracting  Roughness  and  Specular  Sharpness  for  Surfaces  with  a 
Weighted  Lambertian  Albedo 

The  Torrance -Sparrow  model  includes  a  diffuse  component  that  follows  the  lambertian 
model,  a  cosine  function.  Some  surfaces  do  not  have  a  pure  cosine  function  for  the  underly¬ 
ing  lambertian  albedo.  They  have  a  weighted  lambertian  albedo.  For  example; 

/  =  i4cos0  +  C(9O°-0)cos0  +  g^^’^^^~^°  ^  j 

Below  is  a  graph  of  the  lambertian  albedo  with  A  =  100,  and  C  =  0. 1 ,  and  C  =  0.0. 


lambertian 


Between  50  degrees  and  90  degrees,  the  difference  between  the  pure  cosine  and  the 
weighted  cosine  is  small.  As  the  angle  between  the  surface  normal  and  the  incident  light 
becomes  smaller,  the  difference  between  the  two  albedos  becomes  larger.  We  have  previ- 
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ously  assumed  a  uniform  albedo  over  the  object’s  surface.  This  is  not  the  case  here.  Extrac¬ 
tion  of  the  lambertian  albedo  (assuming  that  it  did  follow  a  cosine  function)  at  different 
points  on  the  object’s  surface  wUl  yield  different  results. 

The  lambertian  albedo  impacts  the  calculations  in  a  number  of  different  ways.  The  calcula¬ 
tion  of  surface  normals,  in  the  three  light  and  four  light  illuminated  regions,  depends  on  the 
accuracy  of  the  lambertian  albedo.  First,  an  error  in  the  lambertian  albedo  will  translate  into 
errors  in  the  accuracy  of  the  surface  normals.  Secondly,  D  equals  the  difference  between  the 
measured  image  brightness  at  each  specular  pixel  and  the  lambertian  intensity.  The  calcula¬ 
tion  of  D  will  be  erroneous  if  the  lambertian  albedo  is  in  error.  Both  of  these  errors,  will 
cause  errors  in  the  extraction  of  the  specular  intensity  and  surface  roughness. 

The  solution  to  this  problem  will  depend,  to  some  extent,  upon  the  specific  geometry  of  the 
imaging  system.  Let  us  assume  a  geometry',  and  see  how  we  can  overcome  the  errors  men¬ 
tioned  above.  Our  camera  will  be  at  (0,0,1).  The  angle  between  the  incident  light  and  our 
camera  axis  will  be  65  degrees.  The  angle  between  the  four  light  sources,  in  the  X/Y  plane, 
will  be  90  degrees.  (The  lights  are  placed  at  the  four  comers  of  a  box.  ) 

With  this  geometry,  the  four  light  illuminated  region  extends  from  40  degree  incident  to  90 
degree  incident.  Within  most  of  this  region,  the  theoretical  cosine  lambertian  albedo  is  close 


to  the  weighted  lambertian  albedo.  Therefore,  we  can  approximate  the  weighted  lambertian 
albedo  with  a  regular  cosine  function  in  this  region.  We  can  estimate  the  magnitude  of  the 
lambertian  albedo  by  computing  the  lambertian  albedo  from  a  100  X  100  pixel  box  within 
the  four  light  region.  So  using  the  cosine  approximation,  we  can  extract  fairly  accurate  sur¬ 
face  normals,  and  compute  a  reasonable  value  for  D,  in  the  four  light  illuminated  region. 
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The  real  problem  exists  in  the  three  light  illuminated  region.  In  this  region,  we  compute  the 


surface  normals  using  the  light  sources  adjacent  to  light  source  3  (light  sources  4  and  2).  The 
angle  between  the  adjacent  light  sources  and  the  three  light  illuminated  region  will  place 
most  of  the  region  within  the  50  degree  to  90  degree  range.  So,  the  surface  normals  com¬ 
puted  with  the  unweighted  lambertian  albedo  (from  the  four  light  illuminated  region),  will 
be  fairly  accurate.  The  value  of  D,  computed  from  the  light  source  3,  wiU  be  in  error.  The 
angle  between  light  source  3  and  the  three  light  illuminated  region  wiU  be  between  0 
degrees  and  about  40  degrees.  At  0  degrees,  a  difference  of  about  10  pixel  intensity  levels 
exists  between  the  weighted  albedo  and  the  unweighted  albedo.  This  is  a  10%  error.  While 
this  error  is  smaU,  it  is  sufficient  to  cause  substantial  errors  in  the  specular  intensity  and  sur¬ 
face  roughness.  The  10  pixel  intensity  error  is  only  10%  of  the  lambertian  brighmess,  but  it 
is  probably  a  much  greater  percentage  of  the  specular  intensity. 


A  specular  reflection  model  which  includes  this  error  is  needed.  We  modify  the  simplified 
Torrance-Sparrow  model  to  include  an  offset  term.: 


(  exp(~Ka~) 

D  =  b\  -3— - ^ 

I  Nz 


+  Offset 


Where  the  offset  is  the  difference  between  the  unweighted  lambertian  albedo  and  the 
weighted  lambertian  albedo.  This  equation  has  three  unknowns,  B,  K,  and  the  Offset.  It  is 
not  possible  to  linearize  this  equation,  in  order  to  extract  K.  The  equation  can  be  solved  by 
simultaneously  searching  for  a  solution  in  three  dimensional  (B  JK.,  Offset)  space. 

We  can  define  the  CHISQR  error  as  the  difference  between  the  actual  value  of  D,  the  differ¬ 
ence  between  the  measured  image  brightness  at  each  specular  pixel  and  the  lambertian 
intensity,  and  the  value  of  D  based  on  the  current  value  of  (B,K,  Offset). 

CHISQR  =  -A{i.j))  -D{B,K,Offset)]- 

‘J 

CHISQR  is  a  measure  of  how  good  the  current  estimate  of  (B,K,  Offset)  fits  the  actual 
image  data.  When  CHISQR  equals  zero  the  model  fits  the  data  exactly.  We  can  search  in  3D 
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CHISQR  space  for  the  minimum  value  of  CHISQR.  This  point  will  give  us  the  best  estimate 
of(B,K,  Offset). 

The  search  direction  can  be  determined  using  a  variety  of  methods  such  as  line  search,  gra¬ 
dient  descent  search,  and  conjugate  gradient  descent  search.  We  chose  gradient  descent.  It 
offers  a  reasonable  trade-off  between  algorithmic  complexity  and  search  efficiency.  We  can 
define  the  gradient  of  CHISQR  in  each  of  the  three  dimensions,  (B,  K,  Offset). 

CHISQR  (  (fi.  K.  OFFSET)  +  8B)  ~  CHISQR  (B.  K.  OFFSET) 

V  bCHISQR  =  - - - - = - 


CHISQR  UB.K.  OFFSET)  +  8K)  -  CHISQR  (B.  K.  OFFSET) 
V  kCHISQR  -  - - 


V  OffseCHISQR 


CHISQR  ({B.K.  OFFSET)  +  SOffsel)  -  CHISQR  (B.  K.  OFFSET) 

bOffsei 


The  gradients  are  normalized  with  respect  to  the  unit  vector  formed  by  the  three  compo¬ 
nents,  (B,  K,  Offset).  Then,  the  negative  of  the  gradient  is  multiplied  by  the  step  size  (5B. 
5K,  SOffset)  and  is  added  to  the  previous  value  of  (B,  K,  Offset)  to  produce  the  new  search 
point. 

The  search  continues  until  the  value  of  CHISQR  reaches  a  minimum. 
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4.8.  Algorithm  Flowchart 


Take  4  images  of  lambenian  sphere. 

Extract  light  scarce  directions. 

Take  4  images  of  object,  linearize  images, 
correct  for  dark  current,  normalize  image  intensities. 


Segment  images  into  4  light  illuminated 
lambertian,  4  light  illuminated  specular, 

3  light  illuminated  regions,  2  light  illuminated  regions, 
and  background. 

Calculate  lambertian  albe  i  from  4  light  illuminated 
lambertian  pixels. 


4  light  illuminated  2  light  illuminated  3  light  illuminated  regions 

lambenian  pixels  regions  ^ 

4  light  ilmminated  I  Calculate  surface  normals 

specular  pixels  ^  using  2  adjacent  light  sources 

Calculate  surface  normals 
with  two  available  lights 


Calculated  surface  normals 
using  3  lights  not  containing 
specu^ity 


Calculate  expected  lambertian 
brightness,  classify  pixels  as 
specular  ^nonspecular. 


Calculate  surface  normals 
using  any  3  lights 


Merge  specular  pixels  from  3  light  and  4  light  regions. 
Check  incident  an^le  for  fresnel  limit 

Perform  gradient  descent  fit  to  simplified 
Torrance-Sparrow  model 
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5.  Results 

5.1.  Simulations: 

Synthesized  images  oi  a  sphere,  including  g  issian  intensity  noise,  were  created  to  test  the 
v..idity  of  our  algorithms. 

5.1.1.  Synthesized  Images 

Four  images  of  a  synthetic  sphere  were  g^^'ierated  with  the  following  light  source  directions; 


light  source  one 
light  source  two 
light  source  three 
light  source  four 
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-.555 

.541 

Image  characteristics  are;  p  =  147,  B  =  50  K  =  16,  o*;  =  0.8.  (The  value  of  cr,  =  0.8  is  the 
average  measured  image  intensity  variance.  Our  images  were  reduced  by  a  factor  of  four  to 
one.  Four  pixels  were  averaged  to  produce  one  resulting  pixel.  The  value  of  a*,  reflects  this.) 
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The  pixel  segmentation  for  the  above  images  is: 


Nonspecular  from  light  4 
Shadow  fromfiight  T 


Shadow  from  light  1  and  2 


Specular  from 
light  4 


Four  light  illuminated 
lambertian  area 


Shadow  from  light 

Specular  fro: 
light  1 


Nonspecular  from  1 , 
Shadow  from  light  j 


Nonspecular  from  light  3, 
Shadow  from  light  1 


Specular  from 
light  3 

,  rea  beyond 
fresnel  limit  of 
incident  angle 
(e,<40°) 


ow  from  light  1  and  4 


Specular  from 
li^ht  2 


% 

Nonspiecular  from  light  2, 
Shadow  from  light  A 


Shadow'  from  light  3  and  4 


The  needle  map  produced  from  the  four  intensity  images  is: 
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The  specular  intensity,  B,  specular  sharpness,  K,  and  the  offset,  Offset,  extracted  for  the  four 
specular  spots  is: 


Offset 

B 

K 

Specular  from  light  source  one 

-0.5 

48.5 

15.9 

Specular  from  light  source  two 

-3,6 

51.2 

14.6 

Specular  from  light  source  three 

-0.3 

48.4 

16.0 

Specular  from  light  source  four 

-0.5 

48.3 

15.9 

Average  values 

-1.2 

49.1 

15.6 
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The  plot  of  intensity  versus  incident  angle  for  light  source  two  is; 
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If  our  data  follows  the  simplified  Torrance-Sparrow  model,  a  plot  of  Ln(D*Nz)  versus 
should  be  linear.  This  can  be  seen  from: 


D  =  B 


exp  ( -KcC) 


Ln{DNz)  =  {-KoT)  +  Ln  [B) 


The  plot  of  a*  versus  ln(D*Nz)  for  light  source  two  is: 

arf<  iiT'^ 
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Ln  (D*Nz) 
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5.2.  Experimental  Results 

The  experimental  results  consist  of  images  of  two  objects,  a  specular  painted  sphere  and  a 
plastic  helmet.  Both  objects  exhibit  a  specular  lobe.  In  addition,  images  of  a  specular  plastic 
bottle  are  included.  The  bottle  exhibits  a  specular  spike.  So,  it  does  not  conform  to  our 
reflectance  model  assumptions,  but  we  are  able  to  extract  the  bottle’s  shape. 

5.2.1.  Experimental  Setup 


Camera 


A  ceiling  mounted,  Sony  XC-57  camera,  with  an  85  mm  Nikkor  lens,  was  used.  The  camera 
to  object  distance  was  approximately  2.5  meters.  Four  ECA,  250  watt,  light  bulbs  were 
mounted  on  light  stands.  The  bulb  to  object  distance  was  approximately  2.6  meters. 

5.2.2.  Painted  Specular  Sphere 

Four  images  of  a  painted,  12.7cm  diameter,  sphere  were  taken.  Light  source  directions  were 
determined  to  be; 
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Raw  image  data  was  linearized,  and  normalized 


f 


The  pixel  segmentation  for  the  above  images  is: 


Shadow  from  light  1 
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Shadow  from  light  2 
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light  4 


Four  light  illuminated 
lambertian  area 
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Specular 
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Nonspecular  from  1 , 
Shadow  from  light  3 
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Shadow  from  light  1 
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The  needle  map,  using  extracted  lambertian  albedo  =139.8,  produced  from  the  four  intensity 
images  is: 
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If  we  increase  the  lambertian  albedo  by  the  Offset,  we  are  able  to  extract  the  surface  normals 
in  the  two  light  illuminated  regions. 
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The  specular  intensity,  B,  specular  sharpness,  K,  and  the  offset.  Offset,  extracted  for  the  four 
specular  spots  is  (The  extracted  lambertian  albedo,  p,  is  135.8.): 


Offset 

B 

K 

Specular  from  light  source  one 

10.1 

43.5 

16.7 

Specular  from  light  source  two 

8.2 

55.1 

17.0 

Specular  from  light  source  three 

14.6 

45.8 

21.2 

Specular  from  light  source  four 

15.1 

39.9 

22.8 

Average  values 

12.0 

46.1 

19.4 
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The  plot  of  measured  intensity  versus  incident  angle  for  light  source  three  is: 


The  plot  of  the  intensity  versus  incident  angle  using  the  extracted  values  of  B,  K,  and  Offset 
for  the  specular  spot  due  to  light  source  three  is  (For  this  plot,  since  the  area  of  interest  is  the 
specular  lobe,  the  lambertian  albedo  is  the  sum  of  Offset  and  p.): 
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The  plot  of  measured  intensity  versus  incident  angle  for  light  source  two  is: 
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One  should  note  the  difference  in  magnitude  between  the  lambertian  albedo’s  cosine  curve 
near  the  specular  part  of  the  intensity  plot,  and  the  intensity  data. 

The  plot  of  the  intensity  versus  incident  angle  using  the  extracted  values  of  B,  K,  and  Offset 
for  the  specular  spot  due  to  light  source  two  is  (For  this  plot,  since  the  area  of  interest  is  the 
sp)ecular  lobe,  the  lambertian  albedo  is  the  sum  of  Offset  and  p.): 
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The  plot  of  a"  versus  ln(D*Nz)  is; 
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One  should  note  the  nonlinearity  of  the  plot.  This  is  caused  by  the  Offset  term  of  our  model. 
When  we  compensate  for  the  Offset,  the  plot  becomes  linear. 

The  plot  of  a-  versus  ln[(D-Offset)*Nzl  is; 
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5.2J.  Plastic  Helmet 

Four  images  were  taken  of  a  plastic  helmet  using  the  following  light  source  directions; 
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The  four  images  contain  interreflections  due  to  concavities  in  the  surface  of  the  helmet. 
Since  our  model  does  not  include  interreflection,  we  manually  segmented  the  images  to 
eliminated  any  areas  where  interreflection  might  occur. 


Light  One  Light  Two  Light  Three  Light  Four 


In  addition  to  interreflection  the  plastic  exhibited  a  small  specular  spike.  Due  to  the  averag¬ 
ing  nature  of  the  pixels  in  the  CCD  array,  most  of  the  images  did  not  show  the  spike.  How¬ 
ever,  the  image  due  to  light  source  number  two  did  show  the  spike.  The  spike’s  presence 
caused  the  gradient  descent  fit  to  not  converge,  for  light  source  number  two’s  specular  lobe. 
To  compensate  for  the  any  spike  in  the  other  images,  we  increased  the  specular  threshold  in 
the  four  light  illuminated  area  to; 


’+  ( J-RI  +  ^R2)^ 
'  d/2  d/2 


With  these  modifications  to  our  algorithm,  we  were  able  to  extract  the  specular  lobe's  char- 
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The  plot  of  or  versus  ln(D*Nz)  for  light  source  four  is: 
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5.2.4.  Plastic  Bottle 

Four  images  were  taken  of  a  plastic  bottle.  The  bottle  exhibits  a  strong  specular  spike.  This 
prevents  us  from  recovering  the  roughness  of  the  bottle.  However,  we  are  able  to  recover  the 
shape  of  the  bottle. 


Light  One 


Light  Two 


Light  Three 


Light  Four 
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pixel  segmentation  of  the  four  intensity  images  is: 
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needle  map  produced  from  the  four  intensity  images  is: 
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6.  Conclusions 

We  have  developed  a  method  for  extracting  the  shap>e  and  roughness  of  surfaces  that  exhibit 
a  specular  lobe.  The  work  includes  three  important  parts:  determination  of  shap>e,  seg  menta¬ 
tion  of  pixels  into  specular  and  nonspecular  pixels,  and  determination  of  surface  roughness. 
Coleman  and  Jain’s  work  for  perfomiing  segmentation  in  areas  illuminated  by  four  lights, 
was  improved  by  making  the  segmentation  statistically  meaningful.  We  develop>ed  methods 
for  recovering  shape  and  for  performing  statistical  segmentation  in  regions  illuminated  by 
three  lights.  Wi  also  dev“Iop)ed  methods  for  recovering  shape  in  regions  illuminated  by  two 
lights.  By  examining  the  illumination  conditions  in  the  different  regions  of  the  gaussian 
sphere,  we  were  able  to  extend  photometric  stereo  to  the  entire  gaussian  sphere.  Finally,  by 
using  a  simplified  version  of  the  Torrance-Sparrow  reflectance  model,  we  were  able  to 
develop  an  algorithm  for  extracting  the  specular  intensity  and  surface  roughness  from  spec¬ 
ular  pixels. 

Our  methods  have  been  shown  to  produce  reasonable  results  mn  synthetic  and  real  images. 
We  have  recovered  the  shape  and  roughness  of  objects  that  exhibit  imperfect  lambertian 
albedos,  and  we  have  also  been  able  to  recover  the  shape  of  objects  that  contain  moderate  to 
large  specular  spikes. 

The  work  has  potential  application  to  inspection  problems  in  industrial  environments.  The 
roughness  extraction  algorithms  can  be  used  to  determine  the  roughness  of  objects  with  a 
specular  lobe.  The  shape  extraction  algorithms  are  fairly  robust,  and  can  be  applied  to 
objects  that  exhibit  a  specular  lobe  or  moderate  specular  spike. 
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8.  Appendix  A 

8.1.  Determining  Surface  Shape  With  Classical  Three  Light  Photometric 
Stereo 

Photometric  Stereo  was  originally  proposed  by  Woodham.  His  method  uses  a  stationary 
camera,  three  light  sources,  and  a  reflectance  map  for  the  surface  being  observed.  The 
reflectance  map  contains  a  transformation  between  surface  orientation  and  image  brighmess 
for  a  given  material  and  light  source  (direction  and  intensity).  For  an  arbitrary  material,  a 
reflectance  map  can  be  built  by  taking  an  image  of  a  sphere  composed  of  the  material  under 
each  of  the  light  sources.  Then  a  simple  transformation  between  the  known  surface  orienta¬ 
tion  of  each  point  on  the  sphere  and  image  intensity  can  be  established  for  each  light  source. 

After  the  reflectance  maps  are  built,  the  shape  of  an  unknown  surface  composed  of  the  same 
material  can  be  determined.  Three  images  of  the  unknown  surface  are  taken  using  the  same 
camera  and  light  sources  that  were  used  to  buUd  the  reflectance  maps.  Then,  a  point  is  found 
in  each  of  the  three  reflectance  maps  that  satisfies  the  measured  intensity  and  possesses  a 
common  surface  orientation.  For  three  images  Il(x,y),  I2(x,y),  13(x,y),  and  their  correspond¬ 
ing  reflectance  maps  Rl(nx,ny,nz),  R2(nx,ny,nz),  R3(nx,ny,nz),  we  need  to  find  a  (nx,ny,nz) 
which  satisfies  the  following  set  of  equations  for  a  given  (x,y). 

I\(x,y)  =  R\(nx,ny,nz) 

12{x,y)  =  jR2(nx.  ny,  nz) 

13  {x.  >•)  =  R3(nx,  ny\  nz) 

For  a  lambertian  (diffuse)  surface,  with  surface  normal,  N,  and  light  source  direction,  S, 


/  =  p  ( SxNx  +  SyNy  +  SzNz) 

where  p  is  the  albedo,  0,  is  the  incident  angle.  (Sx,  Sy,  Sz)  and  (Nx,  Ny,  Nz)  are  the  x.  y,  and 
z  components  of  the  unit  source  vector,  S,  and  unit  surface  nomial  vector,  N,  respectively. 
The  albedo  encompasses  the  light  source  intensity  and  relative  reflectance  of  the  surface.  So, 
for  a  lambertian  surface,  the  intensity  is  just  dependent  on  the  light  source  direction,  surface 
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normal,  and  the  albedo.  The  intensity  is  independent  of  the  viewing  direction. 


Therefore,  if  the  surface  is  lambertian,  we  do  not  need  to  store  an  explicit  reflectance  map. 
We  can  determine  the  surface  normal  of  a  point  if  we  know  the  three  light  source  directions 
and  three  intensity  values  of  a  point.  In  matrix  form  we  can  write: 


LSl.x'  Sly  51^ 
]52jf  Sly  52r; 
!53.r  53 V  53r! 


-I , 


r  -] 
lA'jr! 
I  I 
lA^vi 

!wrl 


where  Six,  Sly,  and  Slz  are  the  x,  y,  and  z  components  of  the  unit  vector  to  light  source 
number  one. 


page  41 


9.  References 

[1]  W.  M.  Silver,  “Determining  Shape  and  Reflectance  Using  Multiple  Images”,  S.M. 
Thesis,  Dept,  of  Electrical  Engineering  and  Computer  Science,  MTT,  Cambridge, 
Massachusetts,  June,  1980. 

[2]  S.  K.  Nayar,  K.  Ikeuchi,  T.  Kanade,  “Surface  Reflections:  Physical  and  Geometrical 
Perspectives”,  IEEE  Trans,  of  Pattern  Analysis  and  Machine  Intelligence,  Vol.  13, 
No.  7,  pp.  611-634,  July,  1991. 

[3]  K.  Torrance  and  E.  Sparrow,  “Theory  for  Off-Specular  Reflection  from  Roughened 
Surlacts”,  Journal  of  the  Optical  Society  of  America,  No.  57,  pp.  1105-1114,  1967 

[4]  R.  J.  Woodham,  “Reflectance  Map  Techniques  for  Analyzing  Surface  Defects  in 
Metal  Castings”,  PhD  Thesis,  Artificial  Intelligence  Laboratory,  MIT,  Cambridge, 
Massachusetts,  June,  1978. 

[5]  K.  Ikeuchi,  “Determining  the  surface  orientations  of  specular  surfaces  by  using  the 
photometric  stereo  method”,  IEEE  Trans,  of  Pattern  Analysis  and  Machine  Intelli¬ 
gence,  Vol.  3,  No.  6,  pp.  661-669,  November,  1981. 

[6]  S.  K.  Nayar,  A.  C.  Sanderson,  L.  E.  Weiss,  D.  D.  Simon,  “Specular  Surface  Inspec¬ 
tion  Using  Structured  Highlight  and  Gaussian  Images”,  IEEE  Trans,  on  Robotics 
and  Automation,  Vol.  6,  No.  2,  pp.  208-218,  April,  1990. 

[7]  E.  N.  Coleman  and  R.  Jain,  “Obtaining  3-dimensional  shape  of  textured  and  specular 
surfaces  using  four-source  photometry”.  Computer  Graphics  and  Image  Processing, 
Vol.  18,  No.  4,  pp.  309-328,  April,  1982. 

[8]  G.  Healey  and  TO.  Binford,  “Local  Shape  from  Specularity”,  Computer  Vision 
Graphics  and  Image  Processing,  Vol.  42,  pp.  62-86,  1988. 

[9]  S.  K.  Nayar,  K.  Ikeuchi,  T.  Kanade,  “Determining  Shape  and  Reflectance  of  Hybrid 
Surfaces  by  Photometric  Sampling”,  IEEE  Trans,  on  Robotics  and  Automation,  Vol. 
6,  No.  4,  pp.  418-431,  August,  1990. 

[10]  K.  Ikeuchi  and  K.  Sato,  “Determining  Reflectance  Parameters  Using  Range  and 
Brightness  Images”,  CMU-CS-90-106,  February,  1990. 

[11]  PR.  Bevington,  “Data  Reduction  and  Error  Analysis  for  the  Physical  Sciences”, 
McGraw-Hill,  New  York,  1969,  pp.  56-65. 


